Prototype-Based Graph Contrastive Learning for Malware Detection
HAN Tong1, WANG Tianfeng1, XU Chengcheng2, GUO Shize1,3, PAN Zhisong1
1. College of Command and Control Engineering, Army Engi-neering University of PLA, Nanjing 210007; 2. College of Computer Science, Sichuan University, Chengdu 610065; 3. School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing 100876
Abstract：Graph contrastive learning can effectively capture the complex structural dependencies in behavior graphs, and now it is an important tool for malware detection. However, existing methods treat all samples indiscriminately or overemphasize hard samples during training, leading to training oscillation and slow convergence. To address this issue, a prototype-based graph contrastive learning method for malware detection is proposed. Curriculum learning is extended from the instance-selection level to the sample-pair level, and the contrastive difficulty of software semantics is defined by jointly considering the typicality of anchors and negative samples. A dual-threshold dynamic scheduler is designed to enable progressive contrastive training from typical cases to edge cases. Furthermore, a history-guided anchor enhancement mechanism that tracks and reweights insufficiently learned hard anchors is introduced, thereby enhancing the representation capability for hard samples. Experiments on four public malware datasets demonstrate that the proposed method significantly improves training stability and convergence speed.
[1] Bilot T, El Madhoun N, Al Agha K, et al. A survey on malware detection with graph representation learning[J/OL]. ACM Computing Surveys A survey on malware detection with graph representation learning[J/OL]. ACM Computing Surveys, 2024, 56(11). https://doi.org/10.1145/366464. [2] Kipf T N, Welling M.Semi-supervised classification with graph convolutional networks[EB/OL]. [2026-06-16]. https://arxiv.org/pdf/1609.02907. [3] Veličković P, Cucurull G, Casanova A, et al. Graph attention networks[EB/OL]. [2026-06-16]. https://arxiv.org/pdf/1710.10903. [4] You Y N, Chen T L, Sui Y D, et al.Graph contrastive learning with augmentations[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2020: 5812-5823. [5] Aryal K, Gupta M, Abdelsalam M, et al. A survey on adversarial attacks for malware analysis[J]. IEEE Access, 2025, 13: 428-459. [6] Wu Y M, Dou S H, Zou D Q, et al. Contrastive learning for robust Android malware familial classification[J/OL]. IEEE Transactions on Dependable Contrastive learning for robust Android malware familial classification[J/OL]. IEEE Transactions on Dependable and Secure Computing, 2022. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9936693. [7] Liu C, Li B, Zhao J, et al. A2-CLM: few-shot malware detection based on adversarial heterogeneous graph augmentation[J]. IEEE Transactions on Information Forensics and Security, 2023, 19: 2023-2038. [8] Bengio Y, Louradour J, Collobert R, et al. Curriculum learning[C]//Proceedings of the 26th Annual International Conference on Machine Learning. New York, USA: ACM, 2009: 41-48. [9] Shokouhinejad H, Razavi-Far R, Higgins G, et al. Enhancing GNN explanations for malware detection with dual subgraph matching[J/OL]. Machine Learning Enhancing GNN explanations for malware detection with dual subgraph matching[J/OL]. Machine Learning and Knowledge Extraction, 2026, 8(1). https://doi.org/10.3390/make8010002. [10] Li H D, Cheng X, Zhao Y J, et al. Understanding model weak-nesses: a path to strengthening DNN-based Android malware detection[J]. Proceedings of the ACM on Software Engineering, 2025, 2: 320-342. [11] Zhou J N, Zeng Z H, Bhat S. CLCL: Non-compositional expre-ssion detection with contrastive learning and curriculum learning[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics(Long Paper). Stroudsburg, USA: ACL, 2023, I: 730-743. [12] Wei C, Wang H Y, Shen W, et al. CO2: consistent contrastive learning with curriculum[EB/OL]. [2026-06-16]. https://openreview.net/pdf?id=U4XLJhqwNF1. [13] Wang X D, Liu Z W, Yu S X.Unsupervised feature learning by cross-level discrimination between instances and groups[EB/OL]. [2026-06-16]. https://arxiv.org/pdf/2008.03813v1. [14] Bayer U, Comparetti P M, Hlauschek C, et al. Scalable, behavior-based malware clustering[EB/OL]. [2026-06-16]. https://sites.cs.ucsb.edu/~chris/research/doc/ndss09_cluster.pdf. [15] Lin J H, Wu C E, Li H R, et al. From prototypes to general distributions: an efficient curriculum for masked image modeling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2025: 20028-20038. [16] Chen Y Z, Ding Z J, Wagner D.Continuous learning for Android malware detection[EB/OL]. [2026-06-16]. https://arxiv.org/pdf/2302.04332. [17] Ke G L, Meng Q, Finley T, et al.LightGBM: a highly efficient gradient boosting decision tree[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 3149-3157. [18] 谢丽霞,魏晨阳,杨宏宇,等.基于多维度动态加权alpha图像融合与特征增强的恶意软件检测方法[J]. 电子学报, 2025, 53(3): 849-863. (Xie L X, Wei C Y, Yang H Y, et al. Malware detection method based on multi-dimensional dynamic weighted alpha image fusion and feature enhancement[J]. Acta Electronica Sinica, 2025, 53(3): 849-863.) [19] Shokouhinejad H, Razavi-Far R, Mohammadian H, et al. Recent advances in malware detection: graph learning and explainability[EB/OL]. [2026-06-16]. https://arxiv.org/pdf/2502.10556. [20] 吴月明,齐蒙,邹德清,等.图卷积网络的抗混淆安卓恶意软件检测[J]. 软件学报, 2023, 34(6): 2526-2542. (Wu Y M, Qi M, Zou D Q, et al. Obfuscation-resilient Android malware detection based on graph convolutional networks[J]. Journal of Software, 2023, 34(6): 2526-2542.) [21] Bu S J, Cho S B.Triplet-trained graph transformer with control flow graph for few-shot malware classification[J/OL]. Information Sciences, 2023, 649. https://doi.org/10.1016/j.ins.2023.119 [22] Liu C, Li B, Zhao J, et al. FewM-HGCL: few-shot malware va-riants detection via heterogeneous graph contrastive learning[J/OL]. IEEE Transactions on Dependable FewM-HGCL: few-shot malware va-riants detection via heterogeneous graph contrastive learning[J/OL]. IEEE Transactions on Dependable and Secure Computing, 2022. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9928211. [23] Gao Y, Hasegawa H, Yamaguchi Y, et al. Unsupervised graph contrastive learning with data augmentation for malware classifica-tion[C]//Proceedings of the 16th International Conference on Emer-ging Security Information, Systems and Technologies. Lisbon, Portugal: IARIA, 2022: 41-47. [24] Qian Y Y, Zhang Y M, Chawla N, et al. Malicious repositories detection with adversarial heterogeneous graph contrastive learning[C]//Proceedings of the 31st ACM International Conference on Information & Knowledge Management. New York, USA: ACM, 2022: 1645-1654. [25] Soviany P, Ionescu R T, Rota P, et al. Curriculum learning: a survey[J]. International Journal of Computer Vision, 2022, 130(6): 1526-1565. [26] Jiang L, Meng D Y, Zhao Q, et al. Self-paced curriculum learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2015, 29(1): 2694-2700. [27] Matiisen T, Oliver A, Cohen T, et al. Teacher-student curriculum learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(9): 3732-3740. [28] Tong Z K, Liang Y X, Ding H H, et al.Directed graph contrastive learning[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2021: 19580-19593. [29] Chu G Y, Wang X, Shi C, et al. CuCo: graph representation with curriculum contrastive learning[C]//Proceedings of the 30th International Joint Conference on Artificial Intelligence. San Francisco, USA: IJCAI, 2021: 2300-2306. [30] Zeng C S, Wang B C, Yuan J H, et al. Multi-task curriculum graph contrastive learning with clustering entropy guidance[C]//Proceedings of the 34th International Joint Conference on Artificial Intelligence. San Francisco, USA: IJCAI, 2025: 6949-6957. [31] Zhao J C, Min W W. SpaICL: image-guided curriculum strategy-based graph contrastive learning for spatial transcriptomics clus-tering[J/OL]. Briefings in Bioinformatics, 2025, 26(4). https://doi.org/10.1093/bib/bbaf433. [32] Fang Y C, Chen C, Zhang W,et al. Prototype learning for adversarial domain adaptation[J/OL]. Pattern Recognition, 2024, 155. https://doi.org/10.1016/j.patcog.2024.110653. [33] You Y N, Chen T L, Shen Y, et al. Graph contrastive learning automated[C]//Proceedings of the 38th International Conference on Machine Learning. San Diego, USA: JMLR, 2021: 12121-12132. [34] Xia J, Wu L R, Chen J T, et al. SimGRACE: a simple framework for graph contrastive learning without data augmentation[C]//Proceedings of the ACM Web Conference. New York, USA: ACM, 2022: 1070-1079. [35] Yin Y H, Wang Q Z, Huang S Y, et al. AutoGCL: automated graph contrastive learning via learnable view generators[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(8): 8892-8900. [36] Chen Y Z, Frías J, Gel Y R.TopoGCL: topological graph contrastive learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(10): 11453-11461. [37] Xu C C, Han T, Wang T F,et al. Semantic-aware contrastive learning for graph classification[J/OL]. Expert Systems with App- lications, 2026, 299(B). https://doi.org/10.1016/j.eswa.2025.129976.
2026, Vol. 39 
